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FOREWORD 

This  research  and  development  was  performed  under  Work  Unit  Z1176-PN-01 
(Improving  the  Navy's  Computer-Managed  Training  System),  as  part  of  an  R&D  project  to 
improve  the  Navy's  operational  computer-managed  instruction  (CMI)  system,  and 
was  sponsored  by  the  Deputy  Chief  of  Naval  Operations  (OP-01). 


'  This  report,  the  first  of  a  series  on  Navy  CMI,  describes  the  problem  areas  that  limit 
the  effectiveness  of  the  CMI  system  and  the  R&D  proposals  that  have  been  developed  to 
address  these  problems.  Future  reports  will  describe  work  being  done  under  these 
proposals.  Results  will  be  used  by  the  Chief  of  Naval  Education  and  Training  (CNET), 
Chief  of  Naval  Technical  Training  (CNTT),  commanding  officers  of  ail  the  Navy  CM! 
schools,  and  others  concerned  with  computer-based  instruction.  - 

Appreciation  is  extended  to  Mr.  Dwain  Chambers  (CNTT  Code  N-824)  (CMI  system 
manager),  the  staff  and  students  at  all  of  the  technical  training  schools  on  the  CMI 
system,  and  the  Service  School  Command,  San  Diego,  for  their  assistance  during  the 
project.  Also,  appreciation  is  extended  to  Dr.  H.  S.  Pennypacker,  Jr.,  University  of  Florida, 
Gainesville,  for  the  consulting  service  he  provided  during  the  problem  analysis  and  initial 
R&D  phases. 


JAMES  F.  KELLY,  JR. 
Commanding  Officer 


JAMES  J.  REGAR 
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SUMMARY 


Background 

The  Navy  is  currently  operating  a  computer-managed  instruction  (CMI)  system 
involving  approximately  9000  students  a  day  in  10  technical  training  schools.  As  the 
system  adds  students,  courses,  and  capabilities,  a  coordinated  R&D  effort  is  required  to 
ensure  the  most  cost-effective  and  productive  technical  training  system. 

Objectives 

The  objectives  of  this  project  were  to  identify  the  problem  areas  that  limit  the 
effectiveness  of  the  CMI  system  and  to  develop  coordinated  R&D  plans  to  solve  problems 
amenable  to  research  solutions. 

Approach 

Major  system  problems  were  identified  and  suitable  research  proposals  were  devel¬ 
oped  through  a  four-phase  effort: 

Phase  I.  Problems  were  identified  by  observing  the  system  and  by  administering 
questionnaires  to  and  holding  structured  interviews  with  CMI  management,  instruction 
staff,  and  student  personnel. 

Phase  11.  Other  computer-based  instruction  systems,  both  civilian  and  military,  were 
examined  to  identify  analogous  problems  and  solutions. 

Phase  Ill.  Candidate  research  proposals  were  developed  to  support  the  major  problem 
areas  identified. 

Phase  IV.  Members  of  the  research  and  operational  technical  training  communities 
assessed  the  proposed  research  plans  in  terms  of  (1)  importance  of  the  problem  involved, 
(2)  feasibility  of  the  proposed  research,  (3)  research  and  human  resources  requirements, 
and  (4)  expected  costs  and  benefits  of  applying  the  product  of  the  R&D. 

Results 


The  following  six  R&D  proposals  were  given  the  highest  priority  by  CNTT  and  CNET, 
and  form  the  basis  of  NAVPERSRANDCEN’s  initial  CMI  R&D  effort: 

1.  Effects  of  incentive  charts  on  rate  of  progress  through  a  CMI  course. 

2.  Instructor  role  in  a  CMI  environment. 

3.  Computer-generated  reports  for  the  management  of  student  learning. 

4.  Development  and  incorporation  of  automated  performance  tests  into  the  CMI 
system. 

5.  Development  of  alternate  test  strategies  to  improve  mastery  and  retention  in 
selected  CMI  courses. 

6.  Development  of  computer  software  to  aid  data  summarization  for  research  and 
management  analysis. 
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INTRODUCTION 


Problem  and  Background 

Development  of  the  Navy  CM!  System 

Since  the  1960s,  the  Navy  Personnel  Research  and  Development  Center  (NAV- 
PERSRANDCEN)  and  its  antecedent  laboratories  have  been  working  with  operational 
Navy  training  commands  to  develop  cost-effective,  computer-based  training  systems  for 
large-scale  operational  training.  In  1968,  the  Naval  Personnel  and  Training  Research 
Laboratory,  San  Diego  (NPTRL)  and  the  Chief  of  Naval  Air  Technical  Training 
(CNATECHTRA)  formally  entered  into  an  advanced  development  project  to  develop  a 
computer-managed  instruction  (CMI)  system.  During  1972,  the  first  course — Aviation 
Fundamentals- -was  officially  conducted  with  the  prototype  system  developed  under  this 
project.  In  1974,  CNATECHTRA  and  the  Chief  of  Naval  Education  and  Training  (CNET) 
adopted  CMI  as  a  formal  component  of  the  Navy  training  system.  In  1975,  the  Navy 
Education  Training  and  Instructional  Systems  Activity  (NETISA)  (now  the  Management 
Information  and  Instructional  Systems  Activity  (MIISA))  let  a  contract  for  automated  data 
processing  (ADP)  hardware  and  software  services  for  the  existing  system.  Since  that 
time,  CNET  and  the  Chief  of  Naval  Technical  Training  (CNTT),  with  NAVPERSRANDCEN 
support,  have  worked  to  develop  courses  and  expand  the  CMI  system.  A  comprehensive 
R&D  project  is  needed  to  support  the  Navy  CMI  system  as  it  expands  in  size  and 
capabilities. 

Physical  Characteristics  of  the  Navy  CMI  System 

The  Navy  CMI  system  is  one  of  the  largest  computer-based  instruction  systems  in 
existence.  This  system,  using  off-line,  individualized  instruction  materials,  currently 
manages  the  daily  instruction  progress  of  about  9000  students  in  10  technical  training 
schools  at  5  locations:  Millington,  TN;  Orlando,  FL;  Great  Lakes,  IL;  San  Diego,  CA;  and 
Pensacola,  FL.  These  schools  teach  the  following  courses:  Basic  Electricity  and 
Electronics,  Radioman  "A,"  Propulsion  Engineering  Basics,  Aviation  Fundamentals,  Avia¬ 
tion  Mechanics,  Avionics  "A,"  and  a  portion  of  the  Electronic  Warfare  Fundamentals 
course.  The  central  computer  facility  for  the  system,  which  is  located  at  MIISA, 
Millington,  TN,  grades  tests,  does  record  keeping,  assigns  study  instructions,  and  provides 
information  to  instructors,  school  directors,  and  the  CMI  system  manager  (ranging  from 
data  about  individual  students  to  summaries  of  system  effectiveness).  The  CMI  system 
will  eventually  expand  to  a  projected  capacity  of  about  15,000  students  in  25  schools  in  6 
locations.  When  fully  operational,  the  system  will  be  responsible  for  the  instruction 
management  of  nearly  30  percent  of  all  the  students  in  Navy  technical  training. 

Although  the  physical  layout  of  the  learning  centers  (LCs)  in  the  Navy  CMI  system 
varies  from  school  to  school,  the  same  components  are  present  in  all  centers.  Generally, 
each  LC  consists  of  a  room  with  rows  of  individual  study  carrels.  Areas  for  the 
distribution  of  instruction  materials  and  equipment  are  located  near  each  LC.  Achieve¬ 
ment  testing  of  the  knowledge  or  theory  portion  of  the  course  may  occur  at  either  the 
student  carrels  or  in  a  designated  test  area.  In  both  cases,  after  the  test  is  completed, 
the  student  submits  a  machine-scorable  answer  sheet  into  an  optical  scanner  (OPSCAN 
Model  17),  and  a  teletypewriter  printer  (GE  Terminet  Model  1200)  provides  feedback  to 
the  student  in  the  form  of  a  Learning  Guide.  In  some  CMI  schools,  special  areas  are  set 
up  for  testing  specific  performance  skills.  For  example,  the  students  may  troubleshoot 
equipment  or  components  that  have  been  prefaulted.  One  instructor  in  each  LC  is 
responsible  for  25  to  30  students,  who  may  seek  assistance  at  any  time  throughout  the 
training  day.  The  total  training  day  is  8  hours,  with  6  hours  typically  spent  in  CMI 
instruction  and  2  hours  in  other  military  training.  To  accommodate  large  numbers  of 


students,  some  schools  may  operate  two  or  even  three  consecutive  shifts  of  trainees  a 
day,  obtaining  maximum  benefit  from  the  physical  plant  and  staff.  Since  individual 
instruction  is  given,  and  students  work  at  varying  paces,  study  continues  during  the  entire 
6-hour  shift.  Students  take  breaks  at  their  own  discretion. 

Figures  la  through  If  illustrate  a  typical  Navy  CMI  LC  at  the  Basic  Electricity  and 
Electronics  School,  San  Diego. 

The  CMI  Instruction  Process 


Navy  CMI  focuses  on  student  interactions  with  the  curriculum  materials  (including 
text  matter  and  equipment),  the  instructor,  and  the  computer  system.  Figure  2  shows  the 
possible  interactions  of  system  components  during  the  instruction  process. 

A  student  may  begin  the  CMI  course  at  any  time,  receiving  a  module  study 
assignment  to  be  worked  at  a  self-determined  pace.  Each  module  generally  takes  2  to  5 
hours  to  complete.  The  student  interacts  with  the  curriculum  materials  by  selecting  and 
studying  the  various  instruction  media  or  pieces  of  equipment.  Although  most  instruction 
material  is  printed  text,  sound  slides  and  other  media  are  also  used.  Students  take  a 
"progress  check"  test  (usually  self-scored)  to  determine  whether  they  have  mastered  the 
lesson  materials  before  they  take  the  final  module  test. 

When  the  students  feel  that  they  have  mastered  all  objectives  for  the  module,  they 
take  a  module  knowledge  test  from  a  test  center.  Responses  are  entered  on  a  machine- 
readable  answer  sheet,  which  is  inserted  into  an  optical  scanner.  Responses  are  then 
processed  by  the  scanner  and  transmitted  on  leased  telephone  lines  to  the  central 
computer  at  Millington,  TN.  Within  30-60  seconds,  the  computer  identifies  the  student 
and  the  test,  scores  the  test,  stores  responses  in  a  student  history  file,  determines  the 
next  assignment,  and  transmits  test  results  and  the  next  assignment  back  to  the  student. 
A  typewritten  Learning  Guide,  obtained  from  the  printer  near  the  scanner  (Figure  lc), 
indicates  missed  questions,  lists  lessons  or  objectives  needing  additional  study,  and 
informs  the  student  of  remediation  tests  that  he  must  complete  after  such  study.  After 
the  student  has  mastered  all  module  objectives,  the  next  module  is  assigned. 

On  the  existing  CMI  system,  the  instructor  transmits  performance  test  responses  to 
the  computer  by  completing  a  check  sheet  and  making  an  administrative  entry.  A  more 
automated  procedure  is  currently  being  followed  at  the  Radioman  "A"  school,  where 
automated  teletyping  performance  testing  has  been  developed.  Here,  a  direct  connection 
between  the  student  response  (teletypewriter  keyboard  performance)  and  the  computer 
provides  automatic  evaluation  of  student  performance. 

Students  interact  with  the  instructor  for  a  variety  of  reasons.  For  example,  they  may 
ask  questions  about  course  subject  matter,  request  additional  materials,  interact  for 
personal  reasons  (e.g.,  problems  in  or  out  of  school  that  affect  progress),  or  simply  request 
confirmation  of  their  progress.  To  interact  well  with  students,  instructors  must  have 
technical  competence,  teaching  ability,  and  interpersonal  counselling  skills.  They  must 
handle  administrative  and  subject-matter  queries,  know  the  instruction  materials  and 
equipment,  and  be  able  to  interpret  and  use  the  student  progress  information  available 
from  the  computer  and  school.  This  computerized  progress  information  comes  from  a 
Student  Response  History  file,  which  includes  individual  student  records,  or  from  a  daily 
Learning  Roster,  which  indicates  the  current  state  of  the  instructional  progress  of  all 
students  in  the  instructor's  LC.  The  roster  tells  the  instructor  what  modules  students  are 
studying,  how  long  they  have  been  assigned  specific  modules,  and,  most  important,  how 
the  actual  instructional  progress  of  each  student  compares  with  his  predicted  progress, 
based 
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At  the  CMI  equipment  clustet,  the  student  on  the  right  is  tearing 
off  his  test  results  from  the  printer  after  entering  the  test 
answer  sheet  into  the  optical  scanner  in  the  foreground. 


Figure  |d.  An  area  is  sot  up  where  students  complete  skill-performance  tests 
using  real  equipment. 


Figure  I.  (f\.ntinued). 


Figure  le.  CMI  students  may  <  he-*  k  om  pretaulted  equipment  or  modules  at  a 
central  performance-test  area  where  test  scoring  is  also  performed 


Figure  If.  Students  may  re<  cue  .-.'.m.iikc  from  their  Learning  ('enter 
instructors  at  am  tr  •  airing  the  instructional  day. 


Figure  (•  ei, tinned). 


DIRECT  INTERACTIONS 
WITH  COMPUTER 

- INDIRECT  INTERACTION 


Figure  2.  A  diagram  of  the  relation  of  components  of  the  Navy  CMI  system. 


on  his  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  test  scores,  age,  and  school 
records.  Ideally,  this  actual  versus  predicted  student  progress  information  would  enable 
the  instructor  to  take  the  corrective  action  needed  to  assure  satisfactory  progress  of 
every  student  through  the  course.  Although  self-pacing  occurs  through  the  instructional 
materials,  it  is  expected  that  the  student  will  maintain  a  study  rate  that  will  result  in 
course  completion  within  the  predicted  time,  and  that  the  instructor  will  assist  the 
student  in  maintaining  a  minimum  predicted  rate  of  progress.  These  instruction  progress 
expectations  minimize  training  time  and  facilitate  timely  assignment  of  school  graduates 
to  follow-on  schools  or  fleet  billets. 

Despite  the  current  success  of  the  system  in  managing  student  progress,  projected 
expansion  dictates  the  following  questions: 

1.  Can  improved  instruction  interactions  bring  more  cost-effective  training? 

2.  Can  the  system  expand  in  size  and  capability  without  excessive  cost  growth? 

3.  Can  improvements  in  Navy  CMI  apply  to  other  services  and  to  industry  and 
academia? 

Objectives 

The  objectives  of  this  effort  were  to  identify  the  problem  areas  that  limit  the 
effectiveness  of  the  CMI  system  and  to  develop  coordinated  R&D  to  solve  problems  that 
are  amenable  to  research  solutions. 


APPROACH 


The  project  commenced  in  1977  and  was  conducted  in  the  following  four  phases: 

1.  Identification  of  CMI  system  problems  and  requirements. 

2.  Review  of  other  CMI  systems. 

3.  Development  of  candidate  R&D  proposals. 

4.  Setting  R<5cD  priorities. 

These  phases  are  described  in  the  following  paragraphs. 

Phase  I-- Identification  of  CMI  System  Problems 

The  CMI  system  manager  (CNTT  Code  N-824)  is  responsible  for  maintaining  an 
operational  training  system  for  thousands  of  students.  Since  the  system  manager  is  the 
focal  point  for  complaints  on  system  problems,  he  was  able  to  identify  a  number  of 
problems  that  he  felt  might  be  amenable  to  an  R&D  solution.  Other  CNTT  staff 
members,  representing  training-program  coordinators  and  CNTT  research  organization, 
were  asked  to  give  their  perception  of  CMI  problems. 

User  perceptions  of  CMI  problems  were  obtained  by  administering  questionnaires  on 
system  effectiveness  to  the  students,  instructors,  and  training  officers  at  the  eight 
schools  then  on  the  CMI  system.  All  elements  of  the  CMI  process  and  the  procedures 
followed  by  the  training  schools  were  examined.  Results  showed  that  user  perceptions  of 
problems  were  consistent  with  those  of  the  CNTT  staff.  These  problems  were  primarily 
related  to  the  interactions  of  system  components- -the  student,  the  curriculum,  the 
instructor--and  to  the  requirement  to  maintain  a  flow  of  properly  trained  students  from 
course  to  course  and  out  to  the  fleet. 

Thirteen  specific  problem  areas  were  identified  and  classified  into  three  general 
categories:  (I)  physical  and  environmental  constraints,  (2)  student  and  system 

deficiencies,  and  (3)  system  capability  limitations.  Problems  in  physical  and 
environmental  constraints  were  most  easily  observed,  but  were  not  usually  correctable  by 
research.  For  example,  repair  of  defective  heating  and  cooling  systems  does  not  call  for 
R<3cD. 

The  second  category--student  and  system  deficiencies--was  characterized  by  unique 
problems  students  meet  as  they  progress  through  the  course.  These  difficulties  stem  from 
inadequate  direction  by  either  the  computer  or  instructor,  or  result  from  problems  met  in 
mastering  the  course  materials  or  tests. 

The  third  general  class  of  problems  concerned  the  CMI  system's  limitations.  For 
example,  the  computer  system  could  not  adequately  maintain  a  student's  step-by-step 
progress  record  that  was  easily  retrievable.  Thus,  if  the  student  had  difficulty  in  latter 
stages  of  the  course,  suitable  records  of  past  progress  were  not  readily  available  to  assist 
in  assigning  remedial  instruction.  Failure  to  provide  adequate  learning  within  a  course  is 
most  serious  when  a  student  who  has  completed  the  CMI  course  cannot  demonstrate, 
during  a  follow-on  course  or  the  on-the-job,  skills  he  is  supposed  to  have  acquired. 

Phase  1I-- Review  of  Other  CMI  Systems 

In  the  second  phase,  large-scale  CMI  training  systems  maintained  by  the  other 
military  services,  industry,  and  academia  were  reviewed  to  identify  possible  solutions  for 
the  types  of  problems  identified  in  the  Navy  CMI  system.  A  comprehensive  review  of  all 
computer-based  systems  was  not  attempted.  Rather,  the  review  focused  on  those  systems 
designed  to  manage  large  numbers  of  students  through  individual  instruction  materials. 
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Systems  selected  for  review  included  the  Advanced  Instructional  System  (A1S),  at  Lowry 
Air  Force  Base,  CO;  the  Computerized  Training  System  (CTS)  at  Fort  Gordon,  GA;  and 
civilian  systems  at  the  Universities  of  Illinois,  Pittsburgh,  California  at  Irvine,  and 
Florida.  Results  of  the  reviews  showed  that  virtually  all  of  these  systems  experienced 
similar  problems.  Solutions  were  not  directly  available  and  would  have  to  be  tailored  to 
specific  systems.  The  Navy  system,  it  was  noted,  was  more  advanced  in  some  operational 
respects  than  many  of  the  non-Navy  systems.  Certainly,  the  goals  of  military  training 
made  the  Navy  system  more  capable  than  non-Navy  systems  in  predicting  student 
progress- -despite  the  problems  associated  with  this  feature  of  Navy  CMI. 

Phase  III — Development  of  Candidate  R&D  Proposals 

In  the  third  phase,  NAVPERSRANDCEN  researchers  and  consultants,  with  input  from 
the  schools  and  management  organizations,  developed  candidate  R&D  proposals  j.n  support 
of  the  13  problem  areas  identified  in  Phase  I.  These  plans,  which  are  listed  below, 
described  the  problem,  the  proposed  research  approach,  and  expected  costs  and  benefits 
should  the  research  be  successful. 

1.  Expanding  strategies  and  tactics  for  remediation  in  the  Navy  CMI  system. 

2.  Evaluating  and  monitoring  knowledge  and  skill  retention  within  and  beyond  the 
Navy  CMI  system. 

3.  Development  and  application  of  student  perception  and  motivation  reports. 

4.  Effects  of  incentive  charts  on  rate  of  progress  through  a  CMI  course. 

5.  Study  management-system  implementation  in  an  existing  course  of  instruction. 

6.  Instructor  role  in  a  CMI  environment. 

7.  The  allocation  of  instructors  in  a  CMI  environment. 

8.  Descriptive  and  predictive  measurement  of  performance  in  the  Navy  CMI 
systems. 

9.  Computer-generated  aids  for  the  management  of  student  learning. 

10.  Development  and  incorporation  of  automated  performance  testing  into  the  CMI 
system. 

11.  Effect  of  test-item  response  characteristics  on  mastery  and  retention  in 
selected  CMI  courses. 


12.  Opportunities  for  software  improvements  in  the  Navy  CMI  system. 

13.  Exploiting  computer  technology  to  improve  computer-based  instructional 
management. 


Phase  IV--Settine  R&D  Priorities 


In  the  fourth  phase,  military  and  civilian  researchers  and  members  of  the  operational 
training  and  management  organizations  met  to  discuss  the  severity  of  the  problems, 
feasibility  of  the  proposed  research  solution,  availability  of  research  and  human  resources, 


8 


and  costs  and  benefits  of  the  solutions.  After  each  proposal  was  presented  and  discussed, 
participants  completed  an  open-ended  questionnaire  to  provide  a  summary  of  opinions.  An 
abstract  of  each  proposal  and  a  consensus  on  the  importance  of  each  are  provided  in  the 
appendix. 

After  all  13  proposals  had  been  discussed,  a  final  period  was  devoted  to  omissions, 
priorities,  and  the  sequencing  of  the  various  R&D  elements.  One  omission  was  identified: 
the  development  of  a  resource-allocation  system  to  facilitate  the  assignment  of  students 
and  instructors  to  study  carrels,  study  materials,  test  equipment,  test  settings,  and 
learning  centers.  It  was  suggested  that  a  program  was  needed  to  determine  how  to  get 
maximum  value  from  minimum  resources.  Development  of  this  resource-allocation 
system  was  judged  to  be  an  operational  rather  than  a  research  issue. 


RESULTS 

CMI  Research  Priorities 

The  CNTT  staff  assigned  priority  to  five  R&D  proposals  that  warranted  immediate 
attention.  CNET  (Code  N-5)  endorsed  CNTT's  priorities  and  added  another  proposal  to 
the  list.  Therefore,  NAVPERSRANDCEN's  initial  CMI  research  was  directed  toward  the 
following  six  proposals: 

1.  Effects  of  incentive  charts  on  rate  of  progress  through  a  CMI  course. 

2.  Instructor  role  in  a  CMI  environment. 

3.  Computer-generated  aids  for  the  management  of  student  learning. 

4.  Development  and  incorporation  of  automated  performance  testing  into  the  CMI 
system. 

5.  Development  of  alternate  test  strategies  to  improve  mastery  and  retention  in 
selected  CMI  courses  (modification  of  Plan  //l  1). 

6.  Development  of  computer  software  improvements  to  aid  data  summaries  for 
research  and  management  analysis  (modification  of  Plan  it  12). 

Initial  Research  Efforts 

This  section  describes  the  initial  research  efforts  conducted  on  the  six  R&D  proposals 
receiving  priority  status. 

Effects  of  Incentive  Charts  on  Rate  of  Progress  Through  a  CMI  Course 

Since  student  motivation  is  a  fundamental  problem  in  CMI,  some  kind  of  incentive 
system  is  needed  to  prod  students  into  continuous  activity.  In  this  program,  incentive 
charts  were  used  to  record  progress  made,  encourage  study,  and  reduce  training  time. 

The  incentive  chart  experiment  was  conducted  at  the  BE&E  "A"  school,  Millington, 
TN,  using  a  NAVPERSRANDCEN  IBM  3110  computer  and  civilian  researchers.  The  results 
demonstrated  a  significant  decrease  in  training  time.  The  experimental  group,  which 
received  the  computer-generated  incentive  charts  on  request,  completed  the  course  in  95 
hours,  compared  to  103  hours  for  the  "no  chart"  control  group. 
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If  the  procedures  work  as  well  with  the  operational  CM  I  equipment  and  Navy 
personnel  as  they  did  in  the  experimental  test,  the  system  can  be  used  at  CM  I  schools  by 
merely  "turning  on"  the  charts.  An  operational  test  is  planned  with  Navy  instructors, 
multiple  schools,  and  the  Navy's  central  computer  system.  This  project  is  an  example  of 
an  R<5cD  effort  that  has  the  potential  to  yield  significant  benefits  with  little  implementa¬ 
tion  costs.  If  the  experiment  results  are  repeated,  a  full  training  day  per  student  could  be 
saved  with  a  cost  avoidance  of  over  $1  million  annually. 

Instructor  Role  in  a  CMI  Environment 


A  contract  has  been  let  to  McDonnell-Douglas,  Inc.  to  develop  the  ideal  CMI 
instructor  role-model  for  the  Navy,  Air  Force,  and  a  proposed  Marine  Corps  CMI  systems. 
Because  of  its  cross-service  application,  this  research  was  funded  jointly  by  NAVPERS- 
RANDCEN  and  the  Advanced  Research  Projects  Agency.  The  functions  defined  for  the 
instruction  will  be  validated  empirically  in  the  Navy  CMI  system.  Results  of  this  project 
are  expected  to  be  available  for  the  instructor  training  course  being  developed  by  the 
Naval  Education  and  Training  Support  Center,  Norfolk. 

This  research  represents  the  first  attempt  to  develop  a  CMI  instructor  role  from  a 
theoretical  perspective  followed  by  an  empirical  validation.  Results  from  this  research 
will  impact  on  virtually  every  existing  military  and  civilian  CMI  system.  Completion  of 
the  work  is  expected  in  FY  1981. 

Computer-Generated  Aids  for  the  Management  of  Student  Learning 

Work  in  this  area  has  focused  on  improving  the  system's  capability  to  predict  a 
student's  completion  time.  Although  the  system  now  predicts  total  course  completion 
time  fairly  accurately,  the  prediction  times  for  completion  of  individual  modules  must  be 
improved.  Work  on  prediction  capability  has  focused  on  the  BE&E  course. 

Development  and  Incorporation  of  Automated  Performance  Testing  into  the  CMI 
System 

This  completed  effort  improved  the  efficiency  of  performance-skill  training  by  using 
CMI  to  score  and  analyze  performance  tests  automatically.  Teletyping  instruction  in  the 
Radioman  "A"  school  at  San  Diego  was  chosen  for  this  study  because  manual  scoring 
procedures  were  ted  o,  time  consuming,  and  subject  to  errors.  Some  instructors  spent 
all  day  manually  scoring  typing  tests.  Personnel  from  the  computer  and  instructional 
development  organizations — the  Management  Information  and  Instructional  Systems 
Activity  (MIISA),  and  the  Instructional  Program  Development  Center  (IPDC),  San  Diego, 
with  support  from  NAVPERSRANDCEN,  developed  a  system  for  automated  testing  of 
teletyping  performance;  NAVPERSRANDCEN  developed  a  system  for  providing  diagnostic 
feedback  to  the  student  on  the  types  of  errors  being  made.  With  automated  testing 
procedures,  the  average  time  required  for  students  to  complete  a  typical  three-test  series 
was  cut  from  40  minutes  to  24  minutes.  More  importantly,  the  new  system  reduced 
typing-course  completion  time  and  student  attrition  rates.  These  savings  are  attributed 
to  the  Error  Distribution  Report  (EDR)  provided  by  the  new  system,  which  gives  the 
student  detailed  information  on  areas  where  improvement  is  needed.  During  the  research, 
a  daily  EDR  reduced  course  completion  time  from  an  average  of  85  hours  to  64  hours,  a 
savings  of  almost  3  full  training  days.  The  EDR  also  helped  reduce  attrition  rates  from  25 
to  10  percent  in  the  experimental  samples.  The  automated  testing  system  and  resulting 
EDRs  are  now  in  regular  use  at  the  Radioman  school. 
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Development  of  Alternate  Test  Strategies  to  Improve  Mastery  and  Retention  in  CMI 
Courses 

This  research  is  concerned  with  determining  the  types  of  knowledge  test  items  and 
testing  procedures  that  are  appropriate  for  technical  training  under  computer  manage¬ 
ment.  At  present,  the  CMI  system  is  limited  to  using  multiple-choice  questions.  Answers 
tc  these  test  items  are  fed  into  an  optical  scanner  via  paper  answer  sheet.  Costs  for 
answer  sheets  and  optical  scanners  are  very  high.  While  the  multiple-choice  testing 
format  is  convenient  for  machine  scoring,  its  exclusive  use  may  not  result  in  the 
necessary  level  of  mastery,  since  it  is  simply  a  measure  of  recognition.  In  developing 
instructions,  fill-in  questions  for  testing  recall  of  material  must  be  used  to  satisfy  certain 
types  of  training  objectives. 

In  response  to  such  criticisms,  the  Propulsion  Engineering  (PE)  school,  Great  Lakes, 
developed  a  method  of  converting  fill-in  item  responses  to  a  modified  multiple-choice 
format  that  permitted  machine  scoring.  This  conversion  procedure,  however,  may  have 
decreased  test  accuracy  and  increased  course  completion  time.  During  the  research, 
NAVPERSRANDCEN  found  that  answer  conversion  did  not  decrease  test  scoring  accuracy 
appreciably,  but  did  require  an  average  of  an  extra  half  day  of  training  in  a  22-day  course. 
While  all  question  types  resulted  in  similar  end-of-course  achievement,  fill-in  questions 
without  cues  resulted  in  better  2-week  retention. 

Current  work  on  alternate  test  strategies  involves  implementing  some  of  these 
findings  by  eliminating  the  optical  scanners  and  paper  answer  sheets.  This  approach  would 
reduce  operating  costs  while  improving  the  instructional  capabilities  of  the  system. 

Development  of  Computer  Software  Improvements  to  Aid  Data  Summaries  for 
Research  and  Management  Analysis 

This  work  has  involved  software  to  improve  the  system's  management  and  research 
capabilities.  The  system  must  improve  its  ability  to  use  massive  amounts  of  data  used  to 
track  students  and  evaluate  instructional  materials  and  system  effectiveness. 

Numerous  improvements  have  already  been  made  in  the  system's  ability  to  summarize 
individual  student,  group,  and  other  data  to  facilitate  changes  in  manuals  or  procedures. 
The  data  summaries  are  available  to  school,  management,  or  research  organizations  and 
have  been  applied  by  CNTT  and  MIISA  to  the  operational  system. 

Future  Research  Issues 

Since  the  Navy  CMI  system  has  matured  since  the  initial  research  priorities  were  set, 
and  since  part  of  the  original  R&D  has  been  completed,  the  operational  training 
organization  priorities  have  shifted  to  meet  current  concerns. 

A  CMI  study  conducted  during  the  problem  analysis  compared  alternate  student  to 
instructor  (S/I)  ratios  in  a  CMI  LC  at  the  BE&E  school,  San  Diego.  The  study  investigated 
whether  student  performance  would  suffer  if  the  S/I  ratio  were  increased.  The  results 
suggested  that,  although  the  increased  S/I  ratio  did  not  result  in  performance  decrements, 
training  time  increased  in  some  career  paths.  The  data  revealed  that  the  role  of  the 
instructor  most  definitely  affects  student  performance,  and  suggested  that  altered 
instructor  duties  may  ultimately  lead  to  more  cost-effective  training. 

From  the  S/I  ratio  study,  it  was  apparent  that  a  more  comprehensive  investigation 
was  needed  to  determine  the  most  cost-effective  ratio.  Further  investigation  should 


involve  different  courses,  larger  sample  sizes,  a  wider  range  of  S/1  ratios,  and  other 
variables,  since  different  courses  and  schools  will  undoubtedly  make  different  demands  on 
their  instructors. 


From  the  test  item  study  performed  at  the  PE  school,  two  research  topics  have 
emerged:  (1)  planning  and  evaluation  of  a  test-answer  input  device  (T1D)  that  would 
replace  the  optical  scanner  and  the  paper  answer  sheets,  and  (2)  development  of  proper 
CM  I  testing  and  remediation  strategies  (an  extension  of  current  research). 


In  addition  to  research  essential  to  improving  current  CMI  systems,  planning  for  a 
future-generation  Navy  computer-based  training  system  is  needed.  It  is  obvious  that  this 
planning  must  begin  soon,  even  if  a  major  system  change  is  not  to  occur  for  several  years. 
This  planning  must  include  the  economic  analysis  suggested  by  Orlansky  and  String1  to 
ensure  development  of  a  more  cost  effective  system.  It  is  only  by  pressing  the  research 
questions  and  the  available  instructional  technology  that  the  CMI  system  can  continue  to 
handle  its  full  share  of  the  Navy  training  responsibility. 


‘Orlansky,  3.,  &  String,  3.  Cost-effectiveness  of  computer-based  instruction  in 
military  training  (IDA  Paper  P-13 7%  Arlington,  VA:  Institute  for  Defense  Analyses. 
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APPENDIX 

CANDIDATAE  CMI  RAD  PROPOSAL* 
ABSTRACTS  AND  PLANNING  SESSION  CONSENSUS 


CANDIDATE  CMI  R&D  PROPOSALS: 
ABSTRACTS  AND  PLANNING  SESSION  CONSENSUS 


This  appendix  presents  abstracts  of  the  13  candidate  research  proposals  developed 
during  the  CMI  R&D  Planning  Session,  along  with  a  participant  consensus  of  the 
importance  of  each  plan.  Order  of  presentation  does  not  imply  order  of  importance. 

1.  Expanding  Strategies  and  Tactics  for  Remediation  in  the  Navy  CMI  System 

Abstract.  This  project  considers  the  development  of  remediation  approaches 
applicable  to  the  CMI  system  to  improve  basic  skills  of  trainees,  remedy  skill  deficiencies 
within  a  course  of  instruction,  or  remedy  technical  skill  deficiencies  that  occur  across  a 
sequence  of  CMI  courses. 

Consensus.  Although  remediation  of  basic  skill  deficiencies  is  important, 
techniques  to  help  acquire  technical  skills  have  even  higher  priority.  Research  should 
focus  on  the  identification  and  use  of  remediation  techniques  that  are  the  greatest  aid  to 
learning.  Appropriate  remediation  techniques  would  simplify  control  of  student  learning 
progress  during  a  course  and  would  aid  the  development  of  individualized  CMI  course 
materials. 


2.  Evaluating  and  Monitoring  Knowledge  and  Skill  Retention  Within  and  Beyond 

the  Navy  CMI  System 

Abstract.  The  intent  of  this  project  is  to  determine  how  long  students  retain 
the  knowledge  and  skills  taught  in  the  CMI  courses.  Their  ability  to  retain  knowledge  will 
be  assessed  after  they  have  begun  work  in  the  fleet  or  just  before  or  during  follow-on 
training  courses.  Assessing  knowledge  and  skill  retention  during  a  follow-on  training 
course  will  be  aided  by  computer-managed  testing  programs  that  reflect  the  course-model 
sequencing  and  structure  of  the  original  course.  Intracourse  retention  would  be 
determined  through  similar  computerized  testing  and  an  off-line  assessment  procedure. 

Consensus.  The  quantification  of  the  retention  problem  is  of  great  importance 
to  the  Navy,  but  must  be  handled  appropriately  to  ensure  comparison  with  proper  baseline 
data  when  determining  the  extent  of  the  problem  in  both  CMI  and  non-CMI  learning 
situations.  Extensive  system  design  effort  may  be  needed  for  proper  measurement  of 
retention  along  with  "fleet  enthusiasm"  for  post-training  assessment.  The  product  of  this 
R&D  effort  would  have  tangible  residual  benefits  for  operational  training,  since  the 
programming  would  remain  in  the  computer  system  itself. 

3.  Development  and  Application  of  Student  Perception  ana  Motivation  Reports 

Abstract.  The  purposes  of  this  project  are  to  develop  automated  techniques  for 
determining  student  perceptions  of  the  instructional  environment  and  to  assess  student 
motivation.  The  automated  student  critiques  would  enable  schools  and  training  manage¬ 
ment  to  obtain  summarized  data,  routinely,  on  levels  of  student  motivation  and  on  student 
opinions  of  the  quality  of  instruction. 

Consensus.  There  is  need  for  a  service-wide  effort  that  would  facilitate  the 
systematic  collection  of  subjective  data  from  the  students  about  their  school  experiences. 
It  should  be  possible  to  relate  these  data  to  changes  made  in  the  school.  Use  of  student 
time  in  the  administration  of  the  critiques  should  be  minimized,  however,  and  the 
automated  critiques  should  focus  on  benefits  to  the  school  and  not  on  the  acquisition  of 
basic  research  data. 


A-l 


4.  Effects  of  Incentive  Charts  on  Rate  of  Progress  Through  a  CMI  Course 


Abstract.  This  R&D  involves  development  of  graphic  displays  of  student 
progress  in  a  CMI  course  as  an  incentive  to  study.  Other  incentives  to  study  would  be 
devised  by  school  management  to  reinforce  the  motivation  provided  by  the  displays. 

Consensus.  This  project  is  of  sound  design  and  important  for  the  proper 
management  of  a  CMI  system.  The  effort  would  yield  an  additional  and  innovative  means 
of  measuring  and  motivating  study  progress.  R<5cD  costs  would  be  low.  System-wide  cost 
could  be  estimated  fairly  accurately  following  the  research  effort.  Previous  research 
indicates  that  payoff  from  this  R&D  effort  would  be  high. 

5.  Study-Management  System  Implementation  in  an  Existing  Course  of  Instruction 

Abstract.  Computer-managed  study  techniques,  developed  in  prior  research 
efforts,  would  be  implemented  in  an  existing  lecture-based  course.  The  R&D  intent  would 
be  to  determine  the  feasibility  of  a  low-cost  method  for  individualizing  a  course  focusing 
on  Navy  schools  that  are  not  presently  scheduled  for  full-scale  instructional  development. 

Consensus.  This  project  serves  the  clearly  worthwhile  function  of  developing 
cost-effective  methods  for  individualizing  additional  courses  into  a  CMI  system. 
Selection  of  the  course  and  careful  course  analyses  and  development  are  extremely 
critical  to  this  effort.  Accurate  documentation  of  implementation  activities  and  costs  is 
necessary  to  derive  maximum  benefit  from  the  R&D  effort. 

6.  Instructor  Role  in  a  CMI  Environment 


Abstract.  This  project  broadly  describes  the  R&D  necessary  to  determine  the 
instructor's  role  in  CMI.  Specifically,  the  proposal  calls  for  the  specification  of  CMI 
instructor  activities,  development  of  an  interim  in-service  training  program  for  immedi¬ 
ate  use  in  CMI  schools,  and  development  of  a  formal  training  program  for  candidate 
instructors.  A  critical  aspect  of  this  proposal  is  the  accurate  specification  of  the 
instructor's  role  so  instructor  aids  from  other  R&D  efforts  can  be  exploited. 

Consensus.  Development  of  an  instructor  training  program  that  accurately 
reflects  the  instructor  duties  in  a  CMI  school  is  very  important  for  a  large  operational 
computer-based  training  system.  It  is  clear  that  research  is  needed  for  proper  instructor 
role  definition,  since  the  majority  of  previous  research  has  dealt  with  instructors  in  a 
lecture-based  school  environment.  A  vitally  important  aspect  of  the  research  is  the 
proper  use  of  instructor  evaluation  measures.  Measures  of  instructor-effectiveness  should 
reflect  not  only  direct  observation  of  instructor  activities,  but  also  the  degree  of  student 
success.  This  research  is  particularly  important  in  view  of  the  limitations  placed  on 
initial  selection  of  individuals  for  instructor  duty.  This  project  should  be  broadened  to 
include  the  development  of  a  training  program  for  CMI  school  managers. 

7.  Allocation  of  Instructors  in  a  Computer-Managed  Instructional  Environment 

Abstract.  This  project  will  determine  the  best  CMI  student-to-instructor  ratio 
and  learning-center  configuration.  It  seeks  to  determine  the  ratio  bandwidth  through 
which  managerial  perrogatives  can  vary  before  cost  or  benefit  thresholds  are  exceeded. 

Consensus.  It  is  important  to  establish  the  S/I  bandwidth  so  that  management 
can  make  decisions  on  manning,  but  it  is  also  necessary  to  consider  the  types  of  activities 


assigned  to  instructors  in  the  learning  center.  This  research  must  be  coordinated  with 
Proposal  6,  which  involves  overall  instructor  training. 


8.  Descriptive  and  Predictive  Measurement  of  Performance  in  the  Navy  CMI 

Systems 

Abstract.  This  project  involves  the  development,  test,  and  application  of  a 
technique,  using  test-response  latency,  to  measure  the  student's  mastery  of  course 
material.  Data  on  how  long  students  take  to  master  segments  of  a  course  can  be  recorded 
during  actual  training.  These  data  can  be  built  into  the  computer  to  aid  predictions  if  it  is 
determined  that  the  benefits  will  justify  the  cost. 

Consensus.  This  project  will  develop  more  sensitive  measurement  of  mastery  in 
a  CMI  course.  Time  is  certainly  one  of  the  achievement  variables.  Although  there  is 
minimum  R&D  effort,  the  cost  of  applying  the  results  of  the  R&D  to  the  actual  operating 
system  may  not  be  justified  by  the  benefits.  Cost-effectiveness  must  be  determined  by 
accurate  estimates  during  the  research  phase.  The  use  of  response  latencies  in  measuring 
performance  is  important,  since  the  measure  of  100  percent  mastery  based  solely  on 
number  of  correct  test  items  is  too  insensitive  for  use  in  an  operational  training  system. 

9.  Computer-Generated  Aids  for  the  Measurement  of  Student  Learning 

Abstract.  This  project  examines  the  daily  progress  reports  instructors  receive 
in  the  CMI  learning  center.  Both  the  predictive  capabilities  of  the  regression  equations 
used  in  estimating  student  progress  and  the  format  for  displaying  information  to  the 
instructor  require  improvements. 

Consensus.  This  project  has  high  priority  because  of  an  existing  problem  in  all 
Navy  CMI  schools.  Consideration  should  be  given  to  expanding  the  project  to  include 
improvement  of  the  data  displays  for  the  school  and  system  managers. 

10.  Development  and  Incorporation  of  Automated  Performance  Testing  into  Com¬ 
puter-Managed  Instruction 

Abstract.  This  project  is  intended  to  identify,  automate,  and  incorporate  in  the 
CMI  system  those  performance  skill-testing  activities  that  would  result  in  more  cost- 
effective  training  and  more  accurate  performance  measurement.  The  initial  effort  in  this 
project  involves  the  automation  of  teletypewriter-keyboard  performance  testing  in  the 
RM  "A"  school  at  San  Diego.  The  R&D  will  emphasize  techniques  that  will  facilitate 
automated  CMI  performance  testing  and  will  accelerate  skill  acquisition. 

Consensus.  While  the  general  issue  of  identifying  skills  and  automating  their 
performance  testing  on  a  CMI  system  is  important  and  has  high  priority,  care  should  be 
taken  in  the  case  of  the  RM  school  not  to  redo  extensive  typing  training  research. 

1 1.  Effect  of  Test-Item  Response  Characteristics  on  Mastery  and  Retention  in 

Selected  Computer- Managed  Courses 

Abstract.  This  project  will  determine  test-item  formats  compatible  with  CM! 
that  provide  maximum  instruction  and  measurement  benefits.  This  research  will  examine 
the  test  response  characteristics,  along  with  other  test  item  formats,  currently  in 
question  at  the  Propulsion  Engineering  School  at  Great  Lakes.  The  validity  of  each  test 
response  procedure  will  be  assessed  against  an  independent  performance  test. 


Consensus.  Because  the  proposal  addresses  a  broad  and  recurring  probJem  in 
the  existing  test  system  used  with  Navy  CMI,  the  R&D  effort  should  move  beyond  the 
limitations  imposed  by  simple  5-alternative  multiple-choice  questions.  The  proposal  will 
apply  to  all  CMI  schools,  although  the  focus  is  on  test  response  characteristics  for 
immediate  use.  Clearly,  the  validity  of  any  of  the  response  formats  should  be  carefully 
determined  for  the  particular  subject  matter  being  taught.  This  project  may  well  be  the 
source  of  some  of  the  most  practical  innovations  to  enrich  and  extend  the  existing  Navy 
CMI  system. 

12.  Opportunities  for  Software  Improvements  to  the  Navy  CMI  System 

Abstract.  This  project  is  designed  to  determine  fully  the  users  and  uses  of  the 
data  available  on  the  Navy  CM!  system.  The  data  use  determination  would  focus  on  those 
uses  that  could  be  exploited  by  the  existing  computer  system  in  the  near  future.  Users 
would  include  not  only  the  personnel  from  the  schools  (students,  instructors,  and 
managers),  but  also  the  training  system  managers,  computer  scientists,  and  research 
personnel.  Identified  data  uses  would  be  refined  and  arranged  in  priorities  with  NETISA, 
the  activity  responsible  for  system  analysis  and  programming. 

Consensus.  This  R&D  effort  will  be  useful  to  school,  management,  and  research 
personnel;  successful  completion  is  a  necessary  step  toward  full-scale  use  of  the  results  of 
other  research.  The  project  is  clearly  related  to  data-management  problems  currently 
experienced  by  the  Navy  and  other  CMI  systems,  and  will  be  of  real  benefit  at  little  cost. 
NETISA  personnel  should  be  intimately  involved  with  this  effort  from  the  outset  to 
maximize  system  benefits. 

13.  Exploiting  Computer  Technology  to  Improve  Computer-Based  Instruction 

Management 


Abstract.  This  R&D  will  determine  the  configuration  of  a  second-generation 
computer  system  for  Navy  CMI. 

Consensus.  This  effort  is  a  necessary  R&D  activity  that  will  facilitate  future 
operational  decisions.  While  the  effort  is  very  worthwhile,  experimental  approaches 
besides  the  Delphi  technique  might  be  considered.  R&D  resources  will  be  required,  and 
the  Navy  will  benefit  greatly,  although  the  benefit  will  be  evident  only  when  the  planning 
for  a  subsequent  computer  configuration  is  initiated.  A  secondary  benefit  will  be  the 
establishment  of  a  precedent  and  procedure  for  long-range  planning  in  the  broad  area  of 
military  technical  training. 
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